The Boston startup uses AI to translate and verify legacy software for defense contractors, arguing modernization can't come at the cost of new bugs. Code Metal, a Boston-based startup that uses AI to write code and translate it into other programming languages, just closed a $125 million Series B funding round from new and existing investors. The news comes just a few months after the startup raised $36 million in series A financing led by Accel. Code Metal is part of a new wave of startups aiming to modernize the tech industry by using AI to generate code and translate it across programming languages. One of the questions that persists about AI-assisted code, though, is whether the output is any good--and what the consequences might be if it's not.

We collect 312 programs from various sources, including daily programs from college homework, the international competition (SV -COMP), benchmarks from previous papers (SLING), and programs from real-world software systems (Linux Kernel, GlibC, LiteOS, and Zephyr).

With the rapid development of code LLMs, many popular evaluation benchmarks, such as HumanEval, DS-1000, and MBPP, have emerged to measure the performance of code LLMs with a particular focus on code generation tasks. However, they are insufficient to cover the full range of expected capabilities of code LLMs, which span beyond code generation to answering diverse coding-related questions.

Work partially completed while at Microsoft Research.